Method for audio and video control response and bandwidth adaptation based on network streaming applications and server using the same

ABSTRACT

A method for audio and video control response and bandwidth adaptation based on network streaming applications is disclosed. A server delivers a received control command to a stream controller while a player empties the expired streaming data stored in a buffer thereof. The stream controller controls the input source of a video/audio capturer. The receiver buffer estimator of the receiver estimates whether the amount of the streaming data stored in the buffer is within a safe range and delivers the estimation result and bandwidth variation information to a stream adaptor. If the amount of the stream data is less than a threshold value, the timestamp scalar is adjusted based on the estimation result and the bandwidth variation information, thereby adjusting the consumption rate of the streaming data in the buffer of the player for re-accumulating streaming data of the buffer.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority of Taiwanese Patent Application SerialNo. 097149999, filed on Dec. 22, 2008, the entirety of which isincorporated by reference herein.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to video control, and more particularly to amethod for audio and video control response and bandwidth adaptationbased on network streaming applications and a server using the same.

2. Description of the Related Art

Generally, when a video stream is retrieved via a network, due to thelarge amount of transmission data, a player must buffer some certainamount of streaming data to adapt to bandwidth variations, so thatinterruption or jittering can be prevented while watching the streamingvideo. However, the streaming latency increases as the length ofbuffered data grows, which results in inconvenience for the users whoneeds interaction controls in the streaming applications. For example,when changing TV channels over the network, the waiting time betweenchanges may be excessively long.

As for video surveillance, when the camera detects any special event,the latest frame for that special event must be transmitted over theInternet to the user immediately, and in such case the server shouldactively trigger a video control. In general, the player is required toprepare some buffered data to adapt to the variations of the networkbandwidth. However, buffering data results in latency and that wouldcause a conflict when rapid control reaction time is required. Thus, amethod which can take both video controlling reaction and bandwidthadaptation into consideration is desirable.

Additionally, when playing the streaming, if the bandwidth issubstantially reduced, generally, transmission latency will occur andthe streaming data stored in the data buffer will be exhausted, thus,requiring the player to stop to accumulate streaming data once again.Thus, a method for adjusting for bandwidth variations is desired to keepsmooth playback when streaming.

Thus, the invention provides a method for audio and video controlresponse and bandwidth adaptation based on network streamingapplications, simultaneously achieving management for video controlreactions and network bandwidth adjustments.

BRIEF SUMMARY OF THE INVENTION

Methods for audio and video control response and bandwidth adaptationbased on network streaming applications are provided. One implement of amethod for audio and video control response and bandwidth adaptationbased on network streaming applications comprises the following. Acontrol command received by a server is delivered to the streamcontroller of the server; at the mean while the player empties thestreaming data stored in the buffer either when discovering the expireddata in the buffer by the player itself or when receiving bufferexpiration notification message from the server. The stream controlleris enabled to control the input source of a video/audio capturer andreset the statistics of a receiving buffer estimator in the QoScontroller. Buffer estimator is used to estimates the amount ofstreaming data stored in the player buffer. The estimation result andbandwidth variation information detected by the server are delivered toa stream adaptor. It determines whether the amount of the streaming datastored in the player buffer is less than a threshold value. The streamadaptor is enabled to adjust a timestamp scalar according to theestimation result and the bandwidth variation information if the amountof the stream data is less than the threshold value, thereby adjustingthe consumption rate of the streaming data stored in the player bufferfor re-accumulating streaming data of the buffer indirectly.

In general, in one aspect, the invention provides apparatus, includingan embodiment of a server comprises a video/audio capturer, a QoScontroller, a communicator, a stream controller, and a stream adaptor.The video/audio capturer selects the input source of streaming data. TheQoS controller further comprises a receiving buffer estimator and acongestion detector. After the player sends a control command, thecommunicator of the server receives control commands from the player,forwards to the stream controller. The player can either discover theexpiration of the buffered streaming data by itself and empty its bufferor flush the expired buffered data notified with the buffer expirationmessage sent from the server. When the stream controller receives thecontrol command from the communicator, it controls the input source andstream contents of a video/audio capturer, and resets the receivingbuffer estimator. The stream adaptor further comprises a profileselector, a table of quality profiles, and a timestamp adjuster. Thereceiver buffer estimator estimates the amount of streaming databuffered in the buffer of the player. The stream adaptor retrieves theestimation result and the bandwidth variation information detected bythe congestion detector and determines whether the amount of thestreaming data stored in the buffer of the player is less than athreshold value. The profile selector adjusts the timestamp scalaraccording to the estimation result and the bandwidth variationinformation if the amount of the streaming data is less than thethreshold value, thereby adjusting the consumption rate of the streamingdata stored in the buffer for re-accumulating streaming data of theplayer buffer.

A detailed description is given in the following embodiments withreference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention can be more fully understood by reading the subsequentdetailed descriptions and examples with references made to theaccompanying drawings, wherein:

FIG. 1 is a schematic view of a player of the present invention;

FIG. 2 is a schematic view of a server of the present invention;

FIG. 3 is a schematic view of a stream controller of the presentinvention; and

FIG. 4 is a flowchart of a method for audio and video control responseand bandwidth adaptation based on network streaming applications of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

Several exemplary embodiments of the invention are described withreference to FIGS. 1 through 4, which generally relate to audio andvideo control response and bandwidth adaptation based on networkstreaming application. It is to be understood that the followingdisclosure provides various different embodiments as examples forimplementing different features of the invention. Specific examples ofcomponents and arrangements are described in the following to simplifythe present disclosure. These are, of course, merely examples and arenot intended to be limiting. In addition, the present disclosure mayrepeat reference numerals and/or letters in the various examples. Thisrepetition is for the purpose of simplicity and clarity and does not initself dictate a relationship between the various described embodimentsand/or configurations.

The invention discloses a method for audio and video control responseand bandwidth adaptation based on network streaming application.

An embodiment of the method for audio and video control response andbandwidth adaptation based on network streaming applications isimplemented in a server so that playing of real-time video can beuninterrupted while streaming data can still be re-accumulated andbuffered and optimum bandwidth adaptation can be achieved under changingnetwork bandwidth. Thus, bandwidth adaptation method is improved whileaudio/video control can be responded smoothly and rapidly.

When interactive controls of video is implemented, such as selectinginput sources, switching TV programs, fine tuning video pictures, andcontrolling the lens of a video camera, the amount of transmission datacan be effectively managed by providing the network bandwidth adaptationmethod in the invention of method for audio and video control responseand bandwidth adaptation based on network streaming applications, and sothat the users can be provided with smooth and stable quality of video.

FIG. 1 is a schematic view of a player of the present invention.

The player 100 of the invention comprises a controller 110, a receiver120, a buffer controller 130, a decoder 140, and a renderer 150. Theplayer 100 receives streaming data from the receiver 120 and stores thestreaming data in the buffer 135 to be managed by the buffer controller130. The buffer controller delivers the streaming data to the decoder140 based on timestamp information of the streaming data for decodingand the decoded stream data is played using the renderer 150. When theplayer 100 performs a video control operation, the controller 110transmits a control command to the server 200. At this time, thestreaming data stored in the buffer 135 should have been expired. Thus,the player 100 can ask the controller 130 to flush the expired bufferedstreaming data while the control command is sent. In another embodimentof this disclosed method, the server 200 can also send a bufferexpiration message to notify the player 100 that the buffered streamingdata within some time period are expired and should be flushed.

FIG. 2 is a schematic view of a server of the present invention.

The server 200 of the invention comprises a communicator 201, a streamcontroller 202, a stream sender 203, a video/audio capturer 204, astream encoder 205, a QoS controller 206, a stream adaptor 209, and anaudio re-sampler 213. The QoS controller 206 further comprises areceiving buffer estimator 207 and a congestion detector 208. The streamadaptor 209 further comprises a profile selector 210, a quality profile211, and a timestamp adjuster 212.

The stream controller 202 receives the control command from the player100 via the communicator 201 to directly manage video contents, such asswitching TV programs or input sources (i.e., the Digital Video Disc(DVD) player), and sends a reset command to enable the receiving bufferestimator 207 to reset estimation parameters for the buffer 135 of theplayer 100, thus making correct determinations for processing subsequentquality adjustment of video contents. In some embodiments, thecommunicator 201 also sends a buffer expiration message to the player100 to notify that some expired data should be flushed.

FIG. 3 is a schematic view of a stream controller of the presentinvention.

The stream controller 202 can directly control modules of the server 200comprising setting encoding parameters of the stream encoder 205,setting capture parameters (the brightness and chrominance of an image,for example) of the video/audio capturer 204, or setting an input sourceof the input switcher 301. For example, the input source comprises a TVtuner 302, a video input terminal 303 (such as S-video or Compositevideo), or an external video camera 304. The stream controller 202controls channel switches of the TV tuner 302 and allows the cameramotion controller 305 to perform panning, tilting, and zoomingoperations on that video camera 304.

The QoS controller 206 is used by the server 200 for QoS adaptation. Thecongestion detector 208 of the QoS controller 206 receives feedbackinformation collected by transmission of the stream data by the streamsender 203 to determine whether a network is congested. The QoScontroller enables the stream adaptor 209 to select one of thepredefined settings in the quality profile 211 based on the networkcongestion status to adapt the streaming data to the bandwidth.

The QoS controller 206 also estimates the capability to endure thebandwidth variation for the player 100 using the receiving bufferestimator 207. When the streaming data buffer in the buffer 135 of theplayer 100 is regarded as insufficient, an appropriate adaptation methodwill be selected by the stream adaptor 209.

When the player 100 performs video switching or the network bandwidth isinstable, the length of buffer 135 will be dynamically adjusted. Thereceiving buffer estimator 207 estimates the streaming data lengthbuffered in the buffer 135 of the player 100 based on the transmissionstatus of the network to determine whether adjusting of the videoquality and video timestamp should be performed in order to let theplayer 100 be able to keep playing streaming smoothly. In this way, theserver 200 can estimate the player buffering status withoutinterchanging messages over the network and make this kind oftransmission overhead minimized A formula for the estimation isrepresented by:

R_(buf)(t_(p)) = Min[(∫_(t = 0)^(t_(p))(r(t) − 1) t − T_(sys) − T_(app)), T_(max)],

where t_(p) represents time passes after receiving the control command,T_(max) represents the maximum time length of streaming data should bebuffered in the buffer 135 of the player 120, r(t) represents the valueof timestamp scalar at time t, T_(sys) represents the time length ofunsent video data in the buffer under the system layer, T_(app)represents the delay time of a data queue for an application, andR_(buf)(t_(p)) represents an accumulated time length, by the estimationof the server 200, of the buffer 135 of the player 100 at time t_(p). Inother words, it represents the maximum time the player could endure tokeep playing if in the worst case the network is heavily congested.

Note that the described estimation method is only an embodiment and isnot to be limitative.

The profile selector 210 of the stream adaptor 209 determines the way toadapt the streaming data to network bandwidth according to bandwidthcongestion information retrieved from the QoS controller 206 and thereceiving buffer estimator 207. The predefined settings in the qualityprofile 211 comprise an encoding bit rate, resolution, a frame rate,frame drop option, and the timestamp scalar.

Specifically, when available bandwidth is reduced, encoding bit rate canbe slight decreased first so that the visual quality will not beaffected too much. If the streaming data buffered in the buffer 135 ofthe player 100 is obviously insufficient by the estimation, especiallyafter flushing the buffer due to video control operations, the timestampscalar would be adjusted with higher priority.

If the setting of changing the encoding quality is selected, the profileselector 210 selects, from the quality profile 211, and applies anappropriate setting to the stream encoder 205. If the setting ofdropping frames is selected, the number of frames to be dropped isdetermined. If the setting of adjusting the timestamp scalar isselected, selected parameters are applied to the timestamp adjuster 212for adjusting the timestamp.

The timestamp is referred to by the player 100 for determining the timeat which a video and audio frame should be played. The timestampadjuster 212 adjusts the timestamp interval between two successiveframes according to the timestamp scalar to ensure the video and audiodata can still be played smoothly and synchronously.

As an exemplary embodiment, when the timestamp scalar is adjusted usingthe timestamp adjuster 212, the playing speed of streaming data of theplayer 100 is affected. If the timestamp scalar is notated as r andassumes r=1.1, the timestamp interval between the two successive frameswill be adjusted from 200000 μs, which is before adjusted in a 5frames-per-second video, to 220000 μs after adjustment. Thus, the timeconsumed to play the streaming data with adjusted timestamp intervalbecomes an r multiple of the original timestamp interval. The value oftimestamp scalar r here is only an example and is not to be limitative.

Advantages of using the described adjustment method are as follows. Whena control operation such as switching programs is performed, streamingdata buffered before the control operation is performed becomesmeaningless and is thus flushed, and only small amount of streaming datais still buffered so that fast control response can be achieved. Whenthe timestamp scalar is adjusted using the timestamp adjuster 212,additional sophisticated designs for the player 100 is not requiredsince that the bandwidth adaptation and buffer control can be completedat the server side and the desirable streaming data can still bere-accumulated without stopping playback (when r>1.0, the rate ofaccumulating the streaming data becomes r−1). Further, when bandwidthadaptation for the server 200 is required, it is another innovativemethod can be used for the bandwidth adaptation besides frame dropping,adjusting the encoding bit rate or frame rate directly. Since framedropping may affect the visual quality dramatically and the fineness ofadjusting the frame rate is often limited by the capability andpractical design of the stream encoder 205.

When the timestamp scalar is adjusted using the timestamp adjuster 212,the audio streaming data captured by the video/audio capturer 204 isre-sampled using the audio re-sampler 213 to ensure that sampled pointsof the audio data are uniformly distributed in the time intervals of theadjusted timestamp. Finally, the video data and the re-sampled audiodata are delivered to the stream encoder 205 and are encoded based onthe selected setting from the quality profile 211, thus outputtingencoded frames. The encoded frames are transmitted by the stream sender203 to the player 100 with the adjusted timestamp and feedbackinformation obtained by the stream sender 203 is returned to the QoScontroller 206 for a next determination.

FIG. 4 is a flowchart of a method for audio and video control responseand bandwidth adaptation based on network streaming applications of thepresent invention.

When a communicator receives a control command (a program switchingcommand, for example), the control command is transmitted to a streamcontroller. As for the player, the player empties the receiving bufferif it discovered that the data are expired. (step S401). Besides inanother embodiment, the control event may also be triggered actively bythe server 200 and in such case the player will empty the buffer onreceiving the buffer expiration message, which specifies the range oftime period should expire. The stream controller controls an inputsource of a video/audio capturer and resets a receiving buffer estimatorof a QoS controller (step S402). The receiving buffer estimator of theQoS controller estimates whether the amount of the stream data stored inthe buffer is within a safe range and delivers the estimation result andbandwidth variation information detected by a congestion detector to astream adaptor for QoS adaptation (step S403).

The QoS controller then determines whether there is sufficient streamingdata for the player to keep playing by estimating the amount of databuffered in the player buffer (step S404). If the amount of the streamdata is less than a threshold value (which indicating the amount of thestream data stored in the buffer is insufficient for playing), theprofile selector and the timestamp adjuster of the stream adaptor adjustthe timestamp scalar based on the estimation result obtained from thereceiving buffer estimator and the bandwidth variation informationobtained by the congestion detector, thereby adjusting the consumptionrate of the streaming data in the buffer for re-accumulating streamingdata of the buffer (step S405); Besides, an appropriate setting is alsoselected from the quality profile to adjust the bit rate of thetransmitted streaming data (step S406). When the receiving bufferestimator determines that the amount of the stream data has beenaccumulated to a predetermined amount T_(max) (i.e. a maximum timelength of the buffer of the receiver 120), the timestamp scalar isrestored to a default value (1, for example) (step S407). Thus,interruption or stop for re-accumulating buffer data again due torunning out of buffered streaming data can be prevented by adjusting theconsumption rate of the streaming data in the buffer.

An embodiment of the method for audio and video control response andbandwidth adaptation based on network streaming applications allows fastreaction of video controls that substantially improves user experienceand provides stable transmission quality and bandwidth adaptation. Thus,acceptable video playing quality can still be provided even under thechanging network environment.

Methods and systems of the present disclosure, or certain aspects orportions of embodiments thereof, may take the form of a program code(i.e., instructions) embodied in media, such as floppy diskettes,CD-ROMS, hard drives, firmware, or any other machine-readable storagemedium, wherein, when the program code is loaded into and executed by amachine, such as a computer, the machine becomes an apparatus forpracticing embodiments of the disclosure. The methods and apparatus ofthe present disclosure may also be embodied in the form of a programcode transmitted over some transmission medium, such as electricalwiring or cabling, through fiber optics, or via any other form oftransmission, wherein, when the program code is received and loaded intoand executed by a machine, such as a computer, the machine becomes anapparatus for practicing and embodiment of the disclosure. Whenimplemented on a general-purpose processor, the program code combineswith the processor to provide a unique apparatus that operatesanalogously to specific logic circuits.

While the invention has been described by way of example and in terms ofthe preferred embodiments, it is to be understood that the invention isnot limited to the disclosed embodiments. To the contrary, it isintended to cover various modifications and similar arrangements (aswould be apparent to those skilled in the art). Therefore, the scope ofthe appended claims should be accorded the broadest interpretation so asto encompass all such modifications and similar arrangements.

1. A method for audio and video control response and bandwidthadaptation based on network streaming applications, comprising:transmitting a control command received by a stream communicator of theserver to a stream controller of the server; the player can eitherdiscover the expiration of the buffered streaming data by itself andempty its buffer or flush the expired buffered data notified with thebuffer expiration message sent by the communicator of the server;enabling the stream controller to control an input source of avideo/audio capturer; estimating whether the amount of streaming datastored in the buffer is within a safe range; delivering the estimationresult and bandwidth variation information detected by the server to astream adaptor; determining whether the amount of the stream data storedin the buffer is less than a threshold value; and enabling the streamadaptor to adjust a timestamp scalar according to the estimation resultand the bandwidth variation information if the amount of the stream datais less than the threshold value, thereby adjusting a consumption rateof the streaming data buffered in the player for re-accumulatingstreaming data of the buffer.
 2. The method for audio and video controlresponse and bandwidth adaptation based on network streamingapplications as claimed in claim 1, wherein the timestamp scalar isrestored to a default value when the streaming data of the buffer isaccumulated to a predetermined amount.
 3. The method for audio and videocontrol response and bandwidth adaptation based on network streamingapplications as claimed in claim 1, wherein one of the settings in aquality profile is selected according to the estimation result and thebandwidth variation information to adjust the bit rate of the streamingdata if the amount of the streaming data buffered in the player is lessthan the threshold value.
 4. A server, comprising: a video/audiocapturer for selecting an input source of stream data; a QoS controller,further comprising a receiving buffer estimator and a congestiondetector; a communicator for receiving a control command from a player;a stream controller for receiving the control command from thecommunicator, enabling the player to empty the expired streaming datastored in a buffer of the player, controlling the input source andstreaming contents of a video/audio capturer, and resetting thereceiving buffer estimator; and a stream adaptor, further comprising aprofile selector, a quality profile, and a timestamp adjuster, whereinthe receiving buffer estimator determines whether the amount of streamdata stored in the buffer of the player is within a safe range, thestream adaptor retrieves the estimation result and bandwidth variationinformation detected by the congestion detector and determines whetherthe amount of the streaming data stored in the buffer of the player isless than a threshold value, and the profile selector adjusts atimestamp scalar according to the estimation result and the bandwidthvariation information if the amount of the stream data is less than thethreshold value, thereby adjusting a consumption rate of the streamingdata stored in the buffer for re-accumulating streaming data of thebuffer.
 5. The server as claimed in claim 4, wherein the receivingbuffer estimator restores the timestamp scalar to a default value whenthe streaming data of the buffer is accumulated to a predeterminedamount.
 6. The server as claimed in claim 4, wherein the profileselector selects one of the settings in a quality profile according tothe estimation result and the bandwidth variation information to adjustbit rate of the streaming data if the amount of the streaming data inthe buffer of the player is less than the threshold value.
 7. The serveras claimed in claim 4, wherein a reset command is sent, when the streamcontroller receives the control command, to enable the receiving bufferestimator to reset a parameter used for determining the buffer of theplayer.
 8. The server as claimed in claim 4, further comprising a streamencoder, wherein the stream controller sets encoding settings of thestream encoder, extracting settings of the video/audio extractor, or aninput source and stream contents of an input switcher.
 9. The server asclaimed in claim 8, further comprising a stream sender, wherein thecongestion detector receives feedback information generated bytransmitting the streaming data by the stream sender to determinewhether a network is congested.
 10. The server as claimed in claim 9,wherein the QoS controller enables the stream adaptor to select one ofthe settings in a quality profile according to the network congestioninformation for adapting bandwidth.
 11. The server as claimed in claim10, wherein the QoS controller estimates the capability to endurebandwidth variation for the player using the receiving buffer estimator.12. The server as claimed in claim 11, wherein the receiving bufferestimator estimates the streaming data length buffered in the player todetermine whether video/audio quality and the timestamp scalar should beadjusted.
 13. The server as claimed in claim 4, wherein the settings inthe quality profile comprise an encoding bit rate, resolution, a framerate, frame drop option, and the timestamp scalar.
 14. The server asclaimed in claim 13, further comprising an audio re-sampler forre-sampling the audio data captured by the video/audio capturer when thetimestamp adjuster has adjusted the timestamp scalar.
 15. The server asclaimed in claim 14, wherein video data of the stream data and there-sampled audio data are delivered to the stream encoder and the outputstreaming data are encoded based on the selected setting from thequality profile.
 16. The server as claimed in claim 15, wherein thestream sender transmits the encoded streaming with timestamp which isadjusted based on the timestamp scalar to the player and the feedbackinformation retrieved by the stream sender is returned to the QoScontroller.
 17. A computer-readable medium encoded with computerexecutable instructions for performing a method for audio and videocontrol response and bandwidth adaptation based on network streamingapplications, wherein the computer executable instructions comprise:transmitting a control command received by stream communicator of theserver to a stream controller of the server to enable a player to emptythe expired streaming data stored in a buffer of the player; enablingthe stream controller to control the input source of a video/audiocapturer; estimating whether the amount of streaming data stored in thebuffer is within a safe range; delivering the estimation result andbandwidth variation information detected by the server to a streamadaptor; determining whether the amount of the streaming data stored inthe buffer is less than a threshold value; and enabling the streamadaptor to adjust the timestamp scalar based on the estimation resultand the bandwidth variation information if the amount of the streamingdata is less than the threshold value, thereby adjusting the consumptionrate of the streaming data buffered in the player for re-accumulatingstreaming data of the buffer.
 18. The computer-readable medium asclaimed in claim 17, wherein the timestamp scalar is restored to adefault value when the streaming data of the buffer is accumulated to apredetermined amount.
 19. The computer-readable medium as claimed inclaim 17, wherein one of the settings in a quality profile is selectedaccording to the estimation result and the bandwidth variationinformation to adjust the bit rate of the streaming data if the amountof the streaming data in the buffer of the player is less than thethreshold value.